Method and system for detecting time domain cardiac parameters by using pupillary response

ABSTRACT

Provided are a method and system for detecting time-domain cardiac information, the method comprising: obtaining moving images of a pupil from a subject; extracting a pupil size variation (PSV) from the moving images; calculating R-peak to R-peak intervals (RRIs) in a predetermined frequency range from the PSV; and obtaining at least one time-domain cardiac parameter by processing the RRIs.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of Korean Patent Application Nos.10-2017-0021521, filed on Feb. 17, 2017, and 10-2017-0147609, filed onNov. 7, 2017, in the Korean Intellectual Property Office, thedisclosures of which are incorporated herein in their entirety byreference.

BACKGROUND 1. Field

One or more embodiments relate to a method of detecting physiologicalinformation by using a pupillary response, and a system using themethod, and more particularly, to method of detecting time-domaincardiac parameters from a pupil size variation, and a system using themethod.

2. Description of the Related Art

In vital signal monitoring (VSM), physiological information can beacquired by a sensor attached to a human body. Such physiologicalinformation includes electrocardiogram (ECG), photo-plethysmograph(PPG), blood pressure (BP), galvanic skin response (GSR), skintemperature (SKT), respiration (RSP) and electroencephalogram (EEG).

The heart and brain are two main organs of the human body and analysisthereof provide the ability to evaluate human behavior and obtaininformation that may be used in response to events and in medicaldiagnosis. The VSM may be applicable in various fields such asubiquitous healthcare (U-healthcare), emotional information andcommunication technology (e-ICT), human factor and ergonomics (HF&E),human computer interfaces (HCIs), and security systems.

Regarding ECG and EEG, sensors attached to the body are used to measurephysiological signals and thus, may cause inconvenience to patients.That is, the human body experiences considerable stress andinconvenience when using sensors to measure such signals. In addition,there are burdens and restrictions with respect to the cost of using theattached sensors and to the movement of the subject, due to attachedsensor hardware.

Therefore, VSM technology is required in the measurement ofphysiological signals by using non-contact, non-invasive, andnon-obtrusive methods while providing unfettered movement at low cost.

Recently, VSM technology has been incorporated into wireless wearabledevices allowing for the development of portable measuring equipment.These portable devices can measure heart rate (HR) and RSP by using VSMembedded into accessories such as watches, bracelets, or glasses.

Wearable device technology is predicted to develop from portable devicesto “attachable” devices shortly. It is also predicted that attachabledevices will be transferred to “edible” devices.

VSM technology has been developed to measure physiological signals byusing non-contact, non-invasive, and non-obtrusive methods that provideunfettered movement at low cost. While VSM will continue to advancetechnologically, innovative vision-based VSM technology is required tobe developed also.

SUMMARY

One or more embodiments include a system and method for inferring anddetecting human vital signs by using a non-invasive and non-obstructivemethod at low cost.

In detail, one or more embodiments include a system and method fordetecting time-domain cardiac parameters by using a pupillary responseor pupil size variation.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

According to one or more exemplary embodiments, the method of detectingtime-domain cardiac parameters, the method comprises obtaining movingimages of a pupil from a subject; extracting a pupil size variation(PSV) from the moving images; calculating R-peak to R-peak intervals(RRI) in a predetermined frequency range from the PSV; and obtaining atleast one time-domain cardiac parameter by processing the RRI.

According to one or more exemplary embodiments, a system adopting themethod of claim 1, the system comprising: a video capturing unitconfigured to capture the moving images of the subject; and a computerarchitecture based analyzing unit, including analysis tools andconfigured to process and analyze the moving images, and calculate theat least one cardiac parameter.

According to one or more exemplary embodiments, the predeterminedfrequency range is a harmonic frequency range of 1/100 of the frequencyrange of an electrocardiogram (ECG) signal obtained by sensors.

According to one or more exemplary embodiments, the predeterminedfrequency range is between 0.005 Hz-0.012 Hz.

According to one or more exemplary embodiments, the at least one cardiacparameter is one of Heart Rate (HR), Standard Deviation of the normal tonormal (SDNN), square root of the mean of the squares of successivenormal RR intervals (rMSSD) and pNN50 (successive normal RR intervals>50ms).

BRIEF DESCRIPTION OF THE DRAWINGS

In these and/or other aspects will become apparent and more readilyappreciated from the following description of the embodiments, taken inconjunction with the accompanying drawings in which:

FIG. 1 shows a procedure for selecting a representative of sound stimuliused in an example test, according to one or more embodiments;

FIG. 2 shows an experimental procedure for measuring the amount ofmovement in an upper body, according to one or more embodiments;

FIG. 3 is a block diagram for explaining an experimental procedure,according to one or more embodiments;

FIG. 4 shows a procedure for detecting a pupil region, according to oneor more embodiments;

FIG. 5 schematically shows processes of obtaining time-domain cardiacparameters from pupil moving images (pupil diameter signals) andelectrocardiogram (ECG) signals;

FIG. 6 shows a sample of averages of amount movement in upper body;

FIG. 7 shows examples of processing for extracting heart rate (HR) fromthe pupillary response and ECG signals;

FIG. 8 shows a comparison example of cardiac time indexes, extractedfrom the pupillary response and ECG signals in movelessness condition(MNC);

FIG. 9 shows a comparison example of the cardiac time index extractedfrom the pupillary response and ECG signals in natural movementcondition (NMC);

FIG. 10 shows an infrared webcam system for capturing pupil images,according to one or more embodiments.

FIG. 11 shows an interface screen of a real-time system, according toone or more embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings, wherein like referencenumerals refer to like elements throughout. In this regard, the presentembodiments may have different forms and should not be construed asbeing limited to the descriptions set forth herein. Accordingly, theembodiments are merely described below, by referring to the figures, toexplain aspects of the present description.

Hereinafter, a method and system for inferring and detectingphysiological signals according to the present inventive concept isdescribed with reference to the accompanying drawings.

The invention may, however, be embodied in many different forms andshould not be construed as being limited to the embodiments set forthherein; rather, these embodiments are provided so that this disclosurewill be thorough and complete, and will fully convey the concept of theinvention to those of ordinary skill in the art. Like reference numeralsin the drawings denote like elements. In the drawings, elements andregions are schematically illustrated. Accordingly, the concept of theinvention is not limited by the relative sizes or distances shown in theattached drawings.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” or “includes” and/or “including” when used in thisspecification, specify the presence of stated features, numbers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, numbers, steps,operations, elements, components, and/or groups thereof.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art and/orthe present application, and will not be interpreted in an overly formalsense unless expressly so defined herein.

The embodiments described below involve processing time-domain cardiacparameters from a pupillary response which is obtained from videoinformation.

The present invention, which may be sufficiently understood through theembodiments described below, involve extraction of time-domain cardiacinformation of a heart from the pupillary response or pupil sizevariation by using a vision system equipped with a video camera such asa webcam without any physical restriction or psychological pressure onthe subject. In particular, the pupillary response is detected from theimage information and time-domain cardiac parameters are extracted fromthe detected pupillary response.

In the experiment of the present invention, the reliability of thetime-domain cardiac parameters extracted from the pupil size variation(PSV) acquired through moving images was compared with the ground truthsignal by ECG sensors.

Experiments in relation to the present invention were performed by usingvideo equipment, and a computer architecture based analyzing system forprocessing and analyzing the moving images, which included analysistools provided by software. The system according to exemplaryembodiments was developed using Visual C++ 2010 and OpenCV 2.4.3. Thesignal processing function for fast Fourier transformation (FFT),band-pass filter (BPF), etc. was provided by LabVIEW 2010.

Experimental Stimuli

In order to cause variations in a physiological state, this experimentused sound stimuli based on the Russell's cir-complex model (Russell,1980). The sound stimuli included a plurality of factors, includingarousal, relaxation, positive, negative, and neutral sounds. The neutralsound was defined by an absence of acoustic stimulus. The steps forselecting sound stimuli are shown in FIG. 1 and listed as follows:

(S11) Nine hundred sound sources were collected from the broadcast mediasuch as advertisements, dramas, and movies.

(S12) The sound sources were then categorized into four groups (i.e.,arousal, relaxation, positive, and negative). Each group was comprisedof 10 commonly selected items based on a focus group discussion for atotal of forty sound stimuli.

(S13) These stimuli were used to conduct surveys for suitability foreach emotion (i.e., A: arousal, R: relaxation, P: positive, and N:negative) based on data gathered from 150 subjects that were evenlysplit into 75 males and 75 females. The mean age was 27.36 years±1.66years. A subjective evaluation was required to select each item for thefour factors, which could result in duplicates of one or more of theitems.

(S14) A chi-square test for goodness-of-fit was performed to determinewhether each emotion sound was equally preferred. Preference for eachemotion sound was equally distributed in the population (arousal: 6items, relaxation: 6 items, positive: 8 items, and negative: 4 items) asshown in Table 1.

Table 1 shows the chi-square test results for goodness-of-fit in whichthe items selected for each emotion are based on comparisons ofobservation and expectation values.

TABLE 1 N Chi-Square Sig. Arousal arousal 1 150 83.867 .000 arousal 2150 45.573 .000 arousal 3 150 58.200 .000 arousal 5 150 83.440 .000arousal 9 150 10.467 .000 arousal 10 150 70.427 .000 Relaxationrelaxation 1 150 131.120 .000 relaxation 2 150 163.227 .000 relaxation 5150 80.720 .000 relaxation 6 150 11.640 .000 relaxation 7 150 82.587.000 relaxation 10 150 228.933 .000 Positive positive 2 150 35.040 .000positive 3 150 90.533 .000 positive 4 150 101.920 .000 positive 5 15066.040 .000 positive 7 150 143.813 .000 positive 8 150 128.027 .000positive 9 150 47.013 .000 positive 10 150 138.053 .000 Negativenegative 1 150 119.920 .000 negative 2 150 59.440 .000 negative 5 150117.360 .000 negative 9 150 62.080 .000

Resurveys of the sound stimuli were conducted in relation to eachemotion from the 150 subjects by using a seven-point scale based on 1indicating strong disagreement to 7 indicating strong agreement.

Valid sounds relating to each emotion were analyzed using principalcomponent analysis (PCA) based on Varimax (orthogonal) rotation. Theanalysis yielded four factors explaining the variance for the entire setof variables. After obtaining the analysis results, representative soundstimuli for each emotion were derived, as shown in Table 2.

In Table 2, the bold type is the same factor, the blur character is thecommunalities <0.5, and the thick, light gray lettering with shading inthe background represents the representative acoustic stimulus for eachemotion.

TABLE 2 Component 1 2 3 4 positive 9 .812 .065 .021 −.033 arousal 9 .751−.353 −.157 .107 relaxation 7 .717 .355 .084 .133 positive 2 .531 −.202.203 .107 positive 3 −.528 .222 .406 −.003 positive 8 .520 .142 .161.074 relaxation 2 .192 .684 .109 .004 relaxation 1 .028 .649 .168 −.147relaxation 5 −.290 .629 −.008 .132 relaxation 6 .025 .628 −.061 .107relaxation 10 .052 .569 −.320 −.187 arousal 10 −.201 .529 −.111 .409positive 10 −.145 .424 .342 −.020 negative 1 −.257 −.009 .672 .123positive 4 .111 .096 .608 −.185 negative 2 −.503 .108 .580 .104 negative9 .289 −.252 .566 −.051 negative 5 .216 −.232 .528 −.094 positive 5 .377.014 .439 −.019 positive 7 .002 .193 .403 .128 arousal 1 −.158 .209−.042 .774 arousal 2 .129 −.049 .015 .765 arousal 5 .210 −.043 .097 .672arousal 3 .566 −.159 −.140 .617

Experimental Procedure

Seventy undergraduate volunteers of both genders, evenly split betweenmales and females, ranging in age from 20 to 30 years old with a mean of24.52 years±0.64 years participated in this experiment. All subjects hadnormal or corrected-to-normal vision (i.e., over 0.8), and no family ormedical history of disease involving visual function, cardiovascularsystem, or the central nervous system. Informed written consent wasobtained from each subject prior to the study. This experimental studywas approved by the Institutional Review Board of Sangmyung University,Seoul, South Korea (2015 Aug. 1).

The experiment was composed of two trials where each trail was conductedfor a duration of 5 min. The first trail was based on the movelessnesscondition (MNC), which involves not moving or speaking. The second trialwas based on a natural movement condition (NMC) involving simpleconversations and slight movements. Participants repeatedly conductedthe two trials and the order was randomized across the subjects. Inorder to verify the difference of movement between the two conditions,this experiment quantitatively measured the amount of movement duringthe experiment by using webcam images of each subject. In the presentinvention, the moving image may include at least one pupil, that is, onepupil or both pupils image.

The images were recorded at 30 frames per second (fps) with a resolutionof 1920×1080 by using a HD Pro C920 camera from Logitech Inc. Themovement measured the upper body and face based on MPEG-4 (Tekalp andOstermann, 2000; JPandzic and Forchheimer, 2002). The movement in theupper body was extracted from the whole image based on framedifferences. The upper body line was not tracking because the backgroundwas stationary.

The movement in the face was extracted from 84 MPEG-4 animation pointsbased on frame differences by using visage SDK 7.4 software from VisageTechnologies Inc. All movement data used the mean value from eachsubject during the experiment and was compared to the difference ofmovement between the two trails, as shown in FIG. 2.

FIG. 2 shows an example of measuring the amount of motion of thesubject's upper body in a state of the face is located at theintersection of the X axis and the Y axis.

In FIG. 2, (A) is an upper body image, (B) is a tracked face image at 84MPEG-4 animation points, (C) and (D) shows the difference between beforeand after frames, (E) is a movement signal from the upper body, and (F)shows movement signals from 84 MPEG-4 animation points.

In order to cause the variation of physiological states, sound stimuliwere presented to the participants during the trails. Each soundstimulus was randomly presented for 1 min for a total of five stimuliover the 5 min trial. A reference stimulus was presented for 3 min priorto the initiation of the task. The detailed experimental procedure isshown in FIG. 3.

The experimental procedure includes the sensor attachment S31, themeasurement task S32 and the sensor removal S33 as shown in FIG. 3, andthe measurement task S32 proceed as follows.

The experiment was conducted indoors with varying illumination caused bysunlight entering through the windows. The participants gazed at a blackwall at a distance of 1.5 m while sitting in a comfortable chair. Soundstimuli were equally presented in both the trials by using earphones.The subjects were asked to constrict their movements and speaking duringthe MNC trial. However, the NMC trial involved a simple conversation andslight movement by the subjects. The subjects were asked to introducethemselves to another person as part of the conversation for soundstimuli thereby involving feelings and thinking of the sound stimuli.During the experiment, an ECG signal and pupil image data were obtained.

ECG signals were sampled and recorded at a 500 Hz sampling rate throughone channel with the lead-I method by an amplifier system including ECG100C amplifiers and a MP100 power supply from BIOPAC System Inc. The ECGsignals were digitized by a NI-DAQ-Pad 9205 of National Instrument Inc.

Pupil images were recorded at 125 fps with a resolution of 960×400 byGS3-U3-23S6M-C infrared camera from Point Grey Research Inc.

Hereinafter, a method for extracting or constructing (recovering) vitalsigns from a pupillary response will be described.

Extraction of a Papillary Response

The pupil detection procedure acquires moving images using the infraredvideo camera system as shown in FIG. 12, and then requires a specificimage processing procedure.

The pupil detection procedure may require following certain imageprocessing steps since the images were captured using an infrared videocamera, as shown in FIG. 4.

FIG. 4 shows a process of detecting a pupil region from the face imageof a subject. In FIG. 4, (A) shows an input image (gray scale) obtainedfrom a subject, (B) shows a binarized image based on an auto threshold,(C) shows pupil positions by the circular edge detection, and (D) showsthe real-time detection result of the pupil region including theinformation about the center coordinates and the diameter of the pupilregion. The threshold value was defined by a linear regression modelthat used a brightness value of the whole image, as shown in Equation 1.Threshold=(−0.118×B _(mean)+1.051×B _(max))+7.973B=Brightnessvalue  <Equation 1>

The next step to determine the pupil position involved processing thebinary image by using a circular edge detection algorithm, as shown inEquation 2 (Daugman, 2004; Lee et al., 2009).

$\begin{matrix}{{{Max}_{({r,x_{0},y_{0}})}{{G\;{\sigma(r)}*\frac{\delta}{\delta\; r}{\oint_{r,x_{0},y_{0}}{\frac{I\left( {x,y} \right)}{2\pi\; r}{ds}}}}}}{{I\left( {x,y} \right)} = {{a\mspace{14mu}{grey}\mspace{14mu}{level}\mspace{14mu}{at}\mspace{14mu}{the}\mspace{14mu}\left( {x,y} \right)\mspace{14mu}{{position}\left( {x_{0},y_{0}} \right)}} = {{center}\mspace{14mu}{position}\mspace{14mu}{of}\mspace{14mu}{pupil}}}}{r = {{radius}\mspace{14mu}{of}\mspace{14mu}{pupil}}}} & \left\langle {{Equation}\mspace{14mu} 2} \right\rangle\end{matrix}$

In case that multiple pupil positions are selected, the reflected lightcaused by the infrared lamp may be used. Then an accurate pupil positionwas obtained, including centroid coordinates (x, y) and a diameter.

Pupil diameter data (signal) was resampled at a frequency range of 1Hz-30 Hz, as shown in Equation 3. The resampling procedure for the pupildiameter data involved a sampling rate of 30 data points, which thencalculated the mean value during 1-s intervals by using a common slidingmoving average technique (i.e., a window size of 1 second and aresolution of 1 second). However, non-tracked pupil diameter data causedby the eye closing was not involved in the resampling procedure.

$\begin{matrix}{{{\left( {SMA}_{m} \right)_{x + n} = \left( \frac{\sum\limits_{i = 1}^{m}P_{i}}{m} \right)_{x}},\left( \frac{\sum\limits_{i = 1}^{m}P_{i}}{m} \right)_{x + 1},\ldots\;,\left( \frac{\sum\limits_{i = 1}^{m}P_{i}}{m} \right)_{x + n}}\mspace{20mu}{{SMA} = {{sliding}\mspace{14mu}{moving}\mspace{14mu}{average}}}\mspace{20mu}{P = {{pupil}\mspace{14mu}{diameter}}}} & \left\langle {{Equation}\mspace{14mu} 3} \right\rangle\end{matrix}$

Detecting Time-Domain Index in Cardiac Activity

The detections of the cardiac time-domain indexes (parameters) are nowdescribed along with FIG. 5. Referring FIG. 5, the parameters of cardiactime-domain include HR, SDNN, rMSSD and pNN50 which is determined andobtained from the pupillary response, and ECG signal (ground truth).

The HR is the interval of heartbeat and is related to the speed ofheartbeat. The BPM is the number of heartbeats for 1 min. It iscalculated from the 60 R-peak to R-peak intervals (RRI). The SDNN is thestandard deviation of the normal to normal intervals of the R-Rintervals. It reflects the ebb and flow of the heart's intrinsicfunction. This measure is indicator of left ventricular dysfunction,peak creatine kinase levels, Killip class, and sudden cardiac death(Casole et al., 1992; McCraty and Atkinson, 1996; Wang and Huang, 2012;Park et al., 2014). The SDNN is highly depressed below 50 ms, andmoderately depressed above 100 ms (McCraty and Atkinson, 1996).

The rMSSD is the square root of the mean of the squares of successivenormal RR intervals. This measure reflects the high frequency(short-term variance) in heart rate variability (HRV), and is anindicator of the regulation of the parasympathetic nervous system (vagalbreak) in the heart (McCraty and Atkinson, 1996; Wang and Huang, 2012;Park et al., 2014).

The pNN50 is the percent (proportion) of successive normal RR intervalsthat differ by more than 50 ms. This measure is closely correlated witha high frequency in the HRV and is an indicator of parasympatheticnervous system control of the HR (Vongpatanasin, et al., 2004; Wang andHuang, 2012).

The resampled pupil diameter data at 1 Hz was processed by the band passfilter (BPF) of 0.005 Hz to 0.012 Hz in order to ensure the informationwas relevant to heart, as shown in Equation (4).

$\begin{matrix}{{{A\left( {j\;\omega} \right)}} = \frac{\omega_{0}\omega}{\sqrt{\left( {\omega_{0}^{2} - \omega^{2}} \right)^{2} + {9\omega^{2}\omega_{0}^{2}}}}} & \left\langle {{Equation}\mspace{14mu} 4} \right\rangle\end{matrix}$

The BPF uses a low pass filter and a high pass filter based on theButterworth filter provided by Labview 2010 (Bogdan, M., & Panu, M.LabVIEW modeling and simulation, of the digital filters—In Engineeringof Modern Electric Systems, 2015 IEEE).

The BPF range of 0.5 Hz-1.2 Hz was related to the cardiac flow andapplied by the harmonic frequency with a 1/100 resolution. Then thefiltered data may be calculated using the PSV based on the framedifference of the pupil diameter. The HR was calculated from the meanvalue of the PSV signals, as shown in Equation (5). The HR means thespeed of the heartbeat was controlled by the ANS as calculated from theRRI in ECG signals (Malik, 1996; McCraty et al., 2009; Park et al.,2014). This procedure may be performed or processed by a sliding windowtechnique (i.e., a window size of 30 s and a resolution of 1 s).

$\begin{matrix}{{{PSV} = \frac{\sum\limits_{i = 1}^{n}{{P_{n + 1} - P_{n}}}}{n}}{{PSV} = {{pupil}\mspace{14mu}{size}\mspace{14mu}{variation}}}{P = {{pupil}\mspace{14mu}{diameter}}}} & \left\langle {{Equation}\mspace{14mu} 5} \right\rangle\end{matrix}$

Other cardiac time domain indexes, such as SDNN, rMSSD, and pNN50, maybe extracted from the RRI signals. The SDNN can be calculated from thestandard deviation of the RRI signals based on a normal range of 0.5 Hzto 1.2 Hz. The rMSSD can be calculated from the square root of the meanof the squares of RRI signals with a normal range. The pNN50 can becalculated from counted number of RR intervals that differ by more than50 ms (Wang and Huang, 2012), as shown in Equation (6).

$\begin{matrix}{{{SDNN} = \sqrt{\frac{1}{N - 1}{\sum\limits_{n = 2}^{N}\left\lbrack {{X(n)} - X} \right\rbrack^{2}}}}{{rMSSD} = \sqrt{\frac{1}{N - 2}{\sum\limits_{n = 3}^{N}\left\lbrack {{X(n)} - {X\left( {n - 1} \right)}} \right\rbrack^{2}}}}{{{pNN}\; 50} = \frac{{NN} < {50\mspace{14mu}{count}}}{{total}\mspace{14mu}{NN}\mspace{14mu}{count}}}} & \left\langle {{Equation}\mspace{14mu} 6} \right\rangle\end{matrix}$

The ECG signals of Lead-I may be recorded at a 500 Hz sampling rate andare processed by the BPF of 0.5 Hz 0.12 Hz. The R-peak may be extractedfrom this recording by using the QRS detection algorithm as discussed byPan and Tompkins (1985). The RRIs are the interval measured from R-peakto R-peak intervals. SDNN, rMSSD, and pNN50 can be calculated fromEquation (6) based on the pupil data as mentioned in the above. Thedetailed procedure for processing the ECG signals is shown in FIG. 5showing a procedure of signal processing of cardiac time index.

Result

The pupillary response was processed to extract the vital signs from thecardiac time domain index, cardiac frequency domain index, EEG spectralindex, and the HEP index of the test subjects. These components werecompared with each index from the sensor signals (i.e., ground truth)based on correlation coefficient (r) and mean error value (ME). The datawas analyzed with respect to both MNC and NMC for the test subjects.

To verify the difference of the amount movement between the twoconditions of MNC and NMC, the movement data was quantitativelyanalyzed. The movement data was a normal distribution based on anormality test of probability-value (p)>0.05, and from an independentt-test. A Bonferroni correction was performed for the derivedstatistical significances (Dunnett, 1955). The statistical significancelevel was controlled based on the number of each individual hypothesis(i.e., α=0.05/n). The statistical significant level of the movement datasat up 0.0167 (upper body, X and Y axis in face, α=0.05/3). The effectsize based on Cohen's d was also calculated to confirm practicalsignificance. In Cohen's d, standard values of 0.10, 0.25, and 0.40 foreffect size are generally regarded as small, medium, and large,respectively (Cohen, 2013).

FIG. 6 shows averages of amount movement in upper body, X and Y axis inface for MNC and NMC (n=140, ***p<0.001) of one subject. Table 3 showsall subjects data of amount movement in upper body, X and Y axis in facefor MNC and NMC.

Referring FIG. 6 and Table 3 according to the analysis, the amount ofmovement in MNC (upper body, X and Y axis for the face) aresignificantly increased compared to the NMC for the upper body(t(138)=−5.121, p=0.000, Cohen's d=1.366 with large effect size), X axisfor the face (t(138)=−6.801, p=0.000, Cohen's d=1.158 with large effectsize), and Y axis for the face (t(138)=−6.255, p=0.000, Cohen's d=1.118with large effect size).

TABLE 3 Movelessness Natural Movement Condition Subjects Condition (MNC)(NMC) Subjects Upper body X axis Y axis Upper body X axis Y axis S10.972675 0.000073 0.000158 1.003305 0.000117 0.000237 S2 0.9610200.000081 0.000170 1.002237 0.000101 0.000243 S3 0.942111 0.0000710.000206 0.945477 0.000081 0.000220 S4 0.955444 0.000067 0.0001890.960506 0.000072 0.000191 S5 0.931979 0.000056 0.000106 0.9720330.000070 0.000153 S6 0.910416 0.000057 0.000103 0.999692 0.0000860.000174 S7 0.862268 0.000055 0.000216 0.867949 0.000071 0.000249 S80.832109 0.000056 0.000182 0.884868 0.000068 0.000277 S9 0.8907710.000099 0.000188 0.890783 0.000099 0.000242 S10 0.869373 0.0000730.000168 0.872451 0.000089 0.000206 S11 0.908724 0.000057 0.0001280.963280 0.000102 0.000187 S12 0.954168 0.000091 0.000180 0.9643220.000181 0.000190 S13 0.846164 0.000070 0.000144 0.917798 0.0000790.000172 S14 0.953219 0.000062 0.000116 1.024050 0.000093 0.000185 S150.936300 0.000068 0.000202 0.952505 0.000101 0.000287 S16 0.9430400.000077 0.000220 0.958412 0.000106 0.000308 S17 0.852292 0.0000990.000199 0.901039 0.000077 0.000310 S18 0.901182 0.000082 0.0002780.920493 0.000084 0.000262 S19 0.943810 0.000075 0.000156 0.9746750.000099 0.000386 S20 0.988983 0.000070 0.000162 1.029716 0.0001750.000184 S21 0.952451 0.000065 0.000102 1.005191 0.000081 0.000141 S220.965017 0.000064 0.000099 0.999090 0.000183 0.000150 S23 1.0688480.000101 0.000200 1.090858 0.000108 0.000255 S24 0.993841 0.0000920.000184 1.052424 0.000111 0.000247 S25 0.883615 0.000064 0.0002580.913927 0.000077 0.000283 S26 0.870531 0.000051 0.000221 0.9065400.000074 0.000252 S27 0.955718 0.000064 0.000126 0.963460 0.0000710.000169 S28 0.968524 0.000061 0.000142 0.985782 0.000075 0.000184 S290.794718 0.000067 0.000119 0.918873 0.000074 0.000136 S30 0.8178180.000064 0.000105 0.914591 0.000073 0.000148 S31 0.937005 0.0000530.000138 0.979654 0.000080 0.000203 S32 0.974895 0.000067 0.0002041.011137 0.000072 0.000215 S33 0.877308 0.000073 0.000134 0.8991940.000087 0.000196 S34 0.867672 0.000063 0.000127 0.894298 0.0000770.000188 S35 0.948874 0.000099 0.000182 0.952532 0.000105 0.000217 S360.968912 0.000109 0.000217 1.020322 0.000115 0.000240 S37 0.8111810.000063 0.000204 0.964774 0.000071 0.000244 S38 0.921204 0.0000610.000160 0.966262 0.000071 0.000213 S39 0.907618 0.000060 0.0001510.951832 0.000076 0.000188 S40 0.907953 0.000061 0.000169 0.9207840.000071 0.000188 S41 0.907145 0.000055 0.000151 0.937417 0.0001710.000196 S42 0.909996 0.000055 0.000163 0.995645 0.000072 0.000222 S430.940886 0.000061 0.000137 0.971473 0.000082 0.000188 S44 0.9791630.000059 0.000127 1.058006 0.000184 0.000244 S45 0.946343 0.0000560.000109 1.029439 0.000082 0.000156 S46 0.951810 0.000061 0.0001540.977621 0.000087 0.000256 S47 0.809073 0.000060 0.000147 0.9613750.000065 0.000252 S48 0.961124 0.000073 0.000176 0.997457 0.0000830.000189 S49 0.994281 0.000074 0.000172 1.020115 0.000094 0.000222 S500.853841 0.000075 0.000194 0.978026 0.000104 0.000247 S51 0.8181710.000059 0.000168 0.850567 0.000091 0.000255 S52 0.845488 0.0000720.000134 0.895100 0.000105 0.000293 S53 0.899975 0.000081 0.0001500.967366 0.000094 0.000179 S54 0.819878 0.000057 0.000106 0.9070990.000108 0.000193 S55 0.824809 0.000061 0.000119 0.854062 0.0000620.000125 S56 0.829834 0.000067 0.000126 0.915019 0.000169 0.000157 S570.836302 0.000066 0.000126 0.892036 0.000083 0.000172 S58 0.8760290.000065 0.000155 0.988827 0.000186 0.000163 S59 0.876581 0.0000650.000149 0.924143 0.000117 0.000296 S60 0.881068 0.000101 0.0002521.063924 0.000109 0.000381 S61 0.880455 0.000055 0.000093 1.0073330.000080 0.000190 S62 0.900065 0.000055 0.000087 1.028052 0.0000760.000176 S63 1.045809 0.000056 0.000102 1.061254 0.000096 0.000161 S641.067929 0.000052 0.000105 1.070771 0.000111 0.000162 S65 0.9499710.000055 0.000101 1.004960 0.000068 0.000143 S66 0.964054 0.0000530.000093 1.068673 0.000169 0.000140 S67 0.828268 0.000054 0.0000820.886462 0.000061 0.000117 S68 0.922679 0.000049 0.000079 0.9452910.000061 0.000102 S69 0.946723 0.000063 0.000112 1.069926 0.0001140.000119 S70 0.977655 0.000064 0.000113 0.999438 0.000065 0.000119 mean0.914217 0.000067 0.000153 0.966343 0.000096 0.000208 SD 0.0615960.000014 0.000044 0.057911 0.000033 0.000058

The time domain index for the cardiac output, HR, BPM, SDNN, rMSSD, andpNN50, were extracted from the pupillary response. These components werecompared with the time domain index from the ECG signals (i.e., groundtruth).

The examples for processing for extracting the HR from the pupillaryresponse and ECG signals are shown in FIG. 7. This experiment was ableto determine the cardiac time indices, HR, BPM, SDNN, rMSSD, and pNN50from the pupillary response by the entrainment of the harmonicfrequency. The cardiac heart rhythm in the range of 0.5 Hz-1.2 Hz wasclosely connected to the circadian pupillary rhythm within the range of0.005 Hz-0.012 Hz. The size variation of the pupil diameter wassynchronized with the heart rhythm where the resolution of harmonicfrequency band is 1/100 resolution. That is, the frequency range is thesame as a harmonic frequency range of 1/100 of the frequency of an ECGsignal (ground truth). Other time domain indexes, BPM, SDNN, rMSSD, andpNN50, were calculated from the HR signals.

In FIG. 7, (A) shows frame difference signals of pupil size, (B) shows awave form by signals of 1 Hz resampled based on sliding moving average(window size: 30 fps and resolution: 30 fps), (C) shows band passfiltered signals of 0.005 Hz-0.012 Hz with harmonic frequency of 1/100f,(D) shows signals of pupil size variation, (E) shows a wave form ofheart rate from pupillary response, (F) shows ECG raw signals, (G) showsdetecting the R-peak (QRS complex) and R-peak to R-peak intervals fromECG raw signals, and (H) shows a wave form of heart rate from ECGsignals (ground truth).

FIG. 8 shows a comparison example of cardiac time indexes (HR, BPM,SDNN, rMSSD, and pNN50) in MNC for a subject, where r=0.921, ME=0.018for HR, r=0.918, ME=1.063 for BPM, r=0.864, ME=1.308 for SDNN, r=0.977,ME=0.100 for rMSSD, r=0.838, ME=1.642 for pNN50.

Comparing the results with the ground truth in the MNC for all subjects,the cardiac time indexes from the pupillary response indicated a strongcorrelation coefficient (r) for all parameters where r=0.898±0.064 forHR; r=0.898±0.064 for BPM; r=0.783±0.088 for SDNN; r=0.944±0.059 forrMSSD; and r=0.804±0.055 for pNN50. All the differences between the meanerror ME of all parameters was low where ME=0.009±0.006 for HR;ME=0.825±0.296 for BPM; ME=3.138±3.453 for SDNN; ME=0.143±0.101 forrMSSD; and ME=1.433±0.346 for pNN50.

This procedure was performed by the sliding window technique where thewindow size was 30 s and the resolution was 1 s by using the recordeddata for 300 s.

Table 4 shows average of correlation coefficient and mean error ofcardiac time index in MNC (N=270, p<0.01). In Table 4, the correlationcoefficient and mean error are the mean value for 70 subjects (in onesubject, N=270).

TABLE 4 Correlation coefficient Mean error Subjects HR BPM SDNN rMSSDpNN50 HR BPM SDNN rMSSD pNN50 S1 0.968 0.968 0.722 0.993 0.818 0.0050.513 1.986 0.055 1.725 S2 0.877 0.877 0.610 0.972 0.830 0.005 0.5751.811 0.045 1.825 S3 0.871 0.871 0.635 0.934 0.850 0.008 0.973 2.2920.142 1.459 S4 0.803 0.803 0.732 0.858 0.793 0.009 1.227 2.866 0.1621.974 S5 0.957 0.957 0.870 0.993 0.773 0.004 0.533 1.133 0.070 1.212 S60.969 0.969 0.900 0.997 0.899 0.004 0.596 1.250 0.079 1.388 S7 0.8660.866 0.682 0.908 0.763 0.007 0.668 1.953 0.090 1.781 S8 0.896 0.8960.830 0.942 0.803 0.004 0.466 1.594 0.049 1.790 S9 0.942 0.942 0.8420.975 0.758 0.006 0.622 2.132 0.063 1.802 S10 0.817 0.817 0.888 0.9890.869 0.008 0.860 3.016 0.253 1.998 S11 0.888 0.888 0.776 0.978 0.8050.015 1.206 3.140 0.079 1.430 S12 0.973 0.973 0.753 0.984 0.764 0.0070.670 5.639 0.393 0.951 S13 0.946 0.946 0.821 0.971 0.814 0.026 1.2374.773 0.382 1.242 S14 0.936 0.936 0.820 0.983 0.727 0.026 1.281 2.5020.062 0.802 S15 0.914 0.914 0.936 0.994 0.868 0.018 1.062 1.397 0.0501.211 S16 0.949 0.949 0.774 0.932 0.817 0.006 0.667 3.560 0.360 0.829S17 0.974 0.974 0.705 0.860 0.731 0.004 0.515 5.968 0.252 0.990 S180.782 0.782 0.721 0.993 0.735 0.023 1.275 1.404 0.093 0.952 S19 0.9660.966 0.929 0.988 0.832 0.005 0.680 0.824 0.091 1.872 S20 0.947 0.9470.862 0.934 0.746 0.005 0.691 2.712 0.090 0.818 S21 0.944 0.944 0.8490.949 0.846 0.006 0.605 2.034 0.071 1.260 S22 0.962 0.962 0.718 0.9930.782 0.006 0.601 1.549 0.042 1.311 S23 0.784 0.784 0.789 0.770 0.7970.009 0.920 5.194 0.151 1.157 S24 0.963 0.963 0.958 0.997 0.770 0.0040.445 1.318 0.116 1.332 S25 0.718 0.718 0.864 0.977 0.838 0.010 0.6301.308 0.100 1.642 S26 0.981 0.981 0.808 0.901 0.701 0.006 1.024 4.9440.262 1.177 S27 0.932 0.932 0.703 0.985 0.782 0.005 0.759 4.185 0.1931.220 S28 0.954 0.954 0.937 0.986 0.793 0.013 0.951 1.965 0.249 1.922S29 0.918 0.918 0.667 0.975 0.815 0.015 1.188 3.783 0.207 1.514 S300.923 0.923 0.850 0.988 0.866 0.013 1.018 1.784 0.100 1.922 S31 0.9540.954 0.734 0.906 0.881 0.008 0.738 4.475 0.148 1.927 S32 0.834 0.8340.794 0.864 0.700 0.013 1.087 27.157 0.402 1.652 S33 0.831 0.831 0.6770.832 0.893 0.005 0.518 2.006 0.046 1.754 S34 0.903 0.903 0.790 0.8890.799 0.005 0.500 2.100 0.052 1.574 S35 0.943 0.943 0.823 0.964 0.8250.005 0.569 1.704 0.040 1.227 S36 0.922 0.922 0.837 0.988 0.800 0.0040.430 1.678 0.054 1.516 S37 0.969 0.969 0.680 0.993 0.856 0.009 1.1601.675 0.027 1.216 S38 0.966 0.966 0.796 0.992 0.718 0.007 0.723 2.6990.181 0.941 S39 0.954 0.954 0.823 0.992 0.794 0.011 1.002 1.655 0.1011.975 S40 0.944 0.944 0.764 0.992 0.817 0.004 0.438 2.659 0.151 1.729S41 0.845 0.845 0.819 0.980 0.807 0.005 0.663 1.292 0.048 1.506 S420.925 0.925 0.834 0.964 0.819 0.020 1.932 7.058 0.375 1.475 S43 0.8760.876 0.826 0.992 0.804 0.013 0.996 2.204 0.201 1.178 S44 0.852 0.8520.677 0.819 0.820 0.013 1.010 6.789 0.364 0.829 S45 0.924 0.924 0.7450.976 0.824 0.008 0.753 2.323 0.120 1.305 S46 0.863 0.863 0.810 0.9460.744 0.006 0.644 1.728 0.074 1.541 S47 0.873 0.873 0.877 0.968 0.8590.011 0.788 3.272 0.145 1.625 S48 0.781 0.781 0.742 0.777 0.888 0.0181.301 6.044 0.264 1.469 S49 0.918 0.918 0.841 0.991 0.882 0.011 0.9691.644 0.186 1.263 S50 0.812 0.812 0.719 0.925 0.888 0.010 0.916 1.8700.162 1.183 S51 0.925 0.925 0.817 0.963 0.800 0.005 0.550 1.311 0.0621.836 S52 0.888 0.888 0.945 0.956 0.728 0.005 0.561 1.294 0.058 0.955S53 0.816 0.816 0.674 0.884 0.761 0.022 1.543 8.718 0.326 1.189 S540.955 0.955 0.730 0.792 0.837 0.005 0.580 8.705 0.203 1.811 S55 0.8800.880 0.888 0.986 0.929 0.006 0.750 1.429 0.042 1.207 S56 0.824 0.8240.654 0.923 0.728 0.016 0.966 1.470 0.077 1.912 S57 0.731 0.731 0.7030.918 0.876 0.020 1.220 6.652 0.220 1.037 S58 0.929 0.929 0.676 0.8630.740 0.009 0.842 6.885 0.283 1.556 S59 0.862 0.862 0.824 0.978 0.7610.010 0.941 1.329 0.146 1.640 S60 0.958 0.958 0.713 0.961 0.813 0.0060.622 1.458 0.171 1.585 S61 0.913 0.913 0.830 0.985 0.700 0.005 0.5451.623 0.064 1.899 S62 0.919 0.919 0.880 0.985 0.830 0.009 0.807 1.3930.044 1.408 S63 0.932 0.932 0.792 0.982 0.751 0.006 0.575 2.005 0.1541.373 S64 0.864 0.864 0.671 0.984 0.706 0.005 0.651 1.882 0.085 1.167S65 0.953 0.953 0.639 0.939 0.884 0.004 0.563 1.180 0.090 0.847 S660.735 0.735 0.943 0.996 0.794 0.004 0.511 0.997 0.080 1.768 S67 0.8970.893 0.611 0.932 0.898 0.009 1.217 1.251 0.035 1.735 S68 0.861 0.8610.771 0.818 0.776 0.006 0.684 3.480 0.167 0.820 S69 0.898 0.898 0.6690.873 0.778 0.009 0.987 2.368 0.111 1.859 S70 0.921 0.918 0.841 0.9450.782 0.018 1.063 2.158 0.071 1.336 mean 0.898 0.898 0.783 0.944 0.8040.009 0.825 3.138 0.143 1.433 SD 0.064 0.064 0.088 0.059 0.055 0.0060.296 3.453 0.101 0.346

FIG. 9 shows a comparison example of the cardiac time index extractedfrom the pupillary response and ECG signals (NMC) for a subject, wherer=0.915, ME=0.007 for HR, r=0.917, ME=0.934 for BPM, r=0.841, ME=3.763for SDNN, r=0.996, ME=0.065 for rMSSD, r=0.636, ME=4.957 for pNN50.

Comparing result with ground truth in NMC for all subjects, the cardiactime index from pupillary response were strong correlation of allparameters with r=0.824±0.091 for HR, r=0.824±0.090 for BPM,r=0.710±0.105 for SDNN, r=0.938±0.077 for rMSSD, and r=0.748±0.082 forpNN50. The difference between the mean error of all parameters was lowwith ME=0.013±0.007 for HR, ME=1.295±0.585 for BPM, ME=4.178±2.501 forSDNN, ME=0.154±0.113 for rMSSD, and ME=1.872±0.979 for pNN50.

This procedure was performed and processed by sliding window technique(window size: 30 s and resolution: 1 s) by using recorded data for 300s. The correlation and mean error were mean value for 70 subjects (inone subject, N=270), as shown in Table 5.

Table 5 shows average of correlation coefficient and mean error ofcardiac time index in NMC (N=270, p<0.01).

TABLE 5 Correlation coefficient Mean error Subjects HR BPM SDNN rMSSDpNN50 HR BPM SDNN rMSSD pNN50 S1 0.760 0.760 0.590 0.940 0.819 0.0141.631 5.792 0.165 1.411 S2 0.819 0.819 0.707 0.958 0.846 0.009 1.0933.647 0.109 1.615 S3 0.923 0.923 0.680 0.765 0.797 0.007 0.958 5.0270.324 3.179 S4 0.780 0.780 0.658 0.955 0.626 0.006 0.886 2.761 0.1211.026 S5 0.801 0.801 0.646 0.906 0.805 0.005 0.807 2.802 0.061 1.248 S60.680 0.680 0.807 0.906 0.738 0.011 1.167 1.872 0.057 2.157 S7 0.5830.583 0.665 0.980 0.737 0.008 0.880 7.492 0.097 2.735 S8 0.948 0.9480.631 0.875 0.688 0.008 0.831 6.002 0.054 2.482 S9 0.874 0.874 0.7700.990 0.636 0.009 0.963 3.763 0.065 4.957 S10 0.744 0.744 0.593 0.9290.684 0.014 1.275 2.677 0.178 2.962 S11 0.929 0.929 0.662 0.989 0.8330.014 1.221 3.819 0.149 1.599 S12 0.951 0.951 0.936 0.994 0.765 0.0251.438 7.297 0.365 2.057 S13 0.664 0.664 0.531 0.897 0.801 0.026 1.5534.627 0.368 3.030 S14 0.816 0.816 0.777 0.971 0.642 0.015 1.258 3.7000.047 3.348 S15 0.795 0.795 0.786 0.988 0.797 0.019 1.583 3.513 0.0492.664 S16 0.890 0.890 0.911 0.984 0.634 0.007 0.936 1.510 0.187 2.986S17 0.915 0.915 0.617 0.983 0.798 0.007 0.937 5.137 0.183 1.681 S180.787 0.787 0.937 0.977 0.868 0.017 1.337 1.045 0.096 2.625 S19 0.7790.779 0.886 0.983 0.602 0.015 1.094 1.497 0.087 3.138 S20 0.915 0.9170.559 0.713 0.620 0.007 0.934 8.762 0.590 3.138 S21 0.932 0.932 0.6740.897 0.836 0.006 0.919 7.140 0.206 2.134 S22 0.929 0.929 0.813 0.9740.805 0.006 0.866 3.100 0.088 1.427 S23 0.634 0.634 0.638 0.954 0.6620.033 3.858 2.985 0.215 0.717 S24 0.822 0.822 0.894 0.995 0.710 0.0171.819 1.458 0.123 3.229 S25 0.842 0.842 0.754 0.985 0.754 0.016 1.7492.211 0.062 1.108 S26 0.871 0.871 0.637 0.993 0.638 0.010 1.118 3.7220.195 2.399 S27 0.822 0.822 0.658 0.976 0.694 0.025 1.377 6.905 0.1012.765 S28 0.751 0.751 0.661 0.765 0.796 0.016 0.949 3.741 0.226 0.267S29 0.948 0.948 0.683 0.976 0.625 0.007 1.192 2.867 0.175 0.884 S300.876 0.876 0.796 0.976 0.628 0.005 0.751 4.662 0.170 1.679 S31 0.8500.850 0.642 0.743 0.847 0.015 1.351 1.505 0.163 2.675 S32 0.878 0.8780.814 0.992 0.708 0.012 1.154 3.122 0.204 0.982 S33 0.945 0.945 0.8160.971 0.853 0.013 1.013 2.415 0.112 1.925 S34 0.737 0.737 0.630 0.9760.730 0.015 1.416 2.769 0.076 0.847 S35 0.812 0.812 0.622 0.966 0.8160.016 1.396 4.748 0.059 1.924 S36 0.714 0.714 0.622 0.745 0.627 0.0101.208 2.266 0.174 0.919 S37 0.828 0.828 0.626 0.839 0.607 0.009 1.0852.464 0.252 1.644 S38 0.747 0.747 0.645 0.758 0.793 0.020 2.090 7.5200.138 2.080 S39 0.869 0.869 0.665 0.984 0.635 0.012 1.190 1.077 0.1391.272 S40 0.684 0.684 0.601 0.987 0.809 0.010 1.134 1.239 0.068 2.917S41 0.733 0.733 0.802 0.814 0.769 0.013 1.623 4.872 0.149 2.989 S420.702 0.702 0.603 0.732 0.774 0.009 1.217 2.570 0.308 0.333 S43 0.8750.875 0.613 0.930 0.668 0.025 3.156 1.341 0.261 2.728 S44 0.691 0.6910.686 0.817 0.773 0.023 2.469 2.685 0.533 0.985 S45 0.855 0.855 0.8680.998 0.847 0.012 1.230 2.694 0.068 0.909 S46 0.883 0.883 0.841 0.9840.749 0.020 2.460 4.043 0.065 1.895 S47 0.964 0.964 0.788 0.986 0.6930.008 0.986 10.771 0.205 0.140 S48 0.821 0.821 0.609 0.973 0.701 0.0090.999 3.852 0.195 0.804 S49 0.656 0.656 0.588 0.923 0.744 0.016 1.44710.696 0.164 1.034 S50 0.890 0.890 0.673 0.966 0.734 0.006 0.808 10.7710.158 1.816 S51 0.847 0.847 0.560 0.983 0.771 0.007 0.850 2.482 0.0833.228 S52 0.885 0.885 0.700 0.983 0.728 0.017 1.296 3.509 0.045 0.228S53 0.753 0.753 0.796 0.994 0.725 0.018 1.364 4.927 0.229 3.327 S540.915 0.915 0.692 0.988 0.828 0.007 0.849 2.041 0.244 1.884 S55 0.7690.769 0.693 0.982 0.868 0.008 1.003 4.094 0.049 2.107 S56 0.869 0.8690.685 0.978 0.737 0.028 1.638 5.756 0.047 1.148 S57 0.639 0.639 0.6630.936 0.642 0.023 1.317 11.910 0.419 0.065 S58 0.788 0.788 0.642 0.9060.870 0.012 1.283 4.372 0.322 1.355 S59 0.774 0.774 0.825 0.995 0.6120.010 1.098 6.318 0.111 1.930 S60 0.784 0.784 0.621 0.961 0.794 0.0070.925 3.221 0.063 1.514 S61 0.754 0.754 0.667 0.978 0.873 0.009 1.0593.310 0.033 1.214 S62 0.858 0.858 0.650 0.955 0.878 0.013 1.353 5.3180.051 3.115 S63 0.949 0.949 0.836 0.973 0.803 0.005 0.717 7.031 0.1351.442 S64 0.938 0.938 0.869 0.996 0.710 0.005 0.634 1.807 0.085 0.222S65 0.878 0.878 0.785 0.994 0.833 0.005 0.695 2.026 0.067 1.412 S660.792 0.792 0.526 0.991 0.832 0.008 0.991 2.073 0.057 1.626 S67 0.8290.829 0.817 0.985 0.619 0.011 1.300 5.141 0.030 2.971 S68 0.894 0.8940.937 0.839 0.812 0.014 1.457 2.895 0.070 2.280 S69 0.909 0.889 0.6830.967 0.853 0.031 3.360 1.461 0.086 1.075 S70 0.942 0.939 0.758 0.9920.780 0.027 0.672 7.803 0.155 1.416 mean 0.824 0.824 0.710 0.938 0.7480.013 1.295 4.178 0.154 1.872 SD 0.091 0.090 0.105 0.077 0.082 0.0070.585 2.501 0.113 0.979

Real-Time System for Detecting the Cardiac Time Domain Parameters

The real-time system for detecting information of the cardiac timedomain was developed based on capturing and processing of pupil images.This system may include an infrared webcam, near IR (Infra-Red light)illuminator (IR lamp) and personal computer for analysis.

The infrared webcam was divided into two types, the fixed type, which isa common USB webcam, and the portable type, which are represented bywearable devices. The webcam was a HD Pro C920 from Logitech Inc.converted into an infrared webcam to detect the pupil area.

The IR filter inside the webcam was removed and an IR passing filterused for cutting visible light from Kodac Inc., was inserted into thewebcam to allow passage of IR wavelength longer than 750 nm, as shown inFIG. 10. The 12-mm lens inside the webcam was replaced with a 3.6-mmlens to allow for focusing on the image when measuring the distance from0.5 m to 1.5 m.

FIG. 10 shows an infrared webcam system for taking pupil images.

The conventional 12 mm lens of the USB webcam shown in FIG. 12 wasreplaced with a 3.6 mm lens so that the subject could be focused when adistance of 0.5 m to 1.5 m was photographed.

FIG. 11 shows an interface screen of a real-time system for detectingand analyzing a biological signal from an infrared webcam and a sensor,where (A): Infrared pupil image (input image), (B): binarized pupilimage, (C): Detecting the pupil area, (D): Output of cardiac timeparameters.

As described in the above, the present invention develops and providesan advanced method for measurements of human vital signs from movingimages of the pupil. Thereby, the measurement of parameters in cardiactime domain can be performed by using a low-cost infrared webcam systemthat monitored pupillary response. The cardiac time index involved HRand other parameters calculated from it such as BPM, SDNN, rMSSD, andpNN50. This result was verified for both the conditions of noise (MNCand NMC) and various physiological states (variation of arousal andvalence level by emotional stimuli of sound) for seventy subjects.

The research for this invention examined the variation in humanphysiological conditions caused by the stimuli of arousal, relaxation,positive, negative, and neutral moods during verification experiments.The method based on pupillary response according to the presentinvention is an advanced technique for vital sign monitoring that canmeasure vital signs in either static or dynamic situations.

The present invention may be applied to various industries such asU-health care, emotional ICT, human factors, HCI, and security thatrequire VSM technology.

It should be understood that embodiments described herein should beconsidered in a descriptive sense only and not for purposes oflimitation. Descriptions of features or aspects within each embodimentshould typically be considered as available for other similar featuresor aspects in other embodiments.

While one or more embodiments have been described with reference to thefigures, it will be understood by those of ordinary skill in the artthat various changes in form and details may be made therein withoutdeparting from the spirit and scope of the disclosure as defined by thefollowing claims.

What is claimed is:
 1. A method of detecting time-domain cardiacinformation, the method comprising: obtaining moving images of a pupilfrom a subject; extracting a pupil size variation (PSV) from the movingimages; calculating R-peak to R-peak intervals (RRI) in a predeterminedfrequency range from the PSV; and obtaining at least one time-domaincardiac parameter by processing the RRI.
 2. The method of claim 1,wherein the predetermined frequency range is a harmonic frequency rangeof 1/100 of the frequency range of an electrocardiogram (ECG) signalobtained by sensors.
 3. The method of claim 2, wherein the predeterminedfrequency range is between 0.005 Hz-0.012 Hz.
 4. The method of claim 2,wherein the at least one cardiac parameter is one of Heart Rate (HR),Standard Deviation of the normal to normal (SDNN), square root of themean of the squares of successive normal RR intervals (rMSSD) and pNN50(successive normal RR intervals>50 ms).
 5. The method of claim 3,wherein the at least one cardiac parameter is one of Heart Rate (HR),Standard Deviation of the normal to normal (SDNN), square root of themean of the squares of successive normal RR intervals (rMSSD) and pNN50(successive normal RR intervals>50 ms).
 6. The method of claim 1,wherein the predetermined frequency range is between 0.005 Hz-0.012 Hz.7. The method of claim 6, wherein the at least one cardiac parameter isone of Heart Rate (HR), Standard Deviation of the normal to normal(SDNN), square root of the mean of the squares of successive normal RRintervals (rMSSD) and pNN50 (successive normal RR intervals>50 ms). 8.The method of claim 1, wherein the at least one cardiac parameter is oneof Heart Rate (HR), Standard Deviation of the normal to normal (SDNN),square root of the mean of the squares of successive normal RR intervals(rMSSD) and pNN50 (successive normal RR intervals>50 ms).
 9. A systemadopting the method of claim 1, the system comprising: a video capturingunit configured to capture the moving images of the subject; and acomputer system including software for performing the method of claim 1,the computer system being configured to process and analyze the movingimages and calculate the at least one cardiac parameter.
 10. The systemof claim 9, wherein the predetermined frequency range is a harmonicfrequency range of 1/100 of the frequency range of an ECG signalobtained by sensors.
 11. The system of claim 10, wherein thepredetermined frequency range is between 0.005 Hz-0.012 Hz.
 12. Thesystem of claim 10, wherein the at least one cardiac parameter is one ofHeart Rate (HR), Standard Deviation of the normal to normal (SDNN),square root of the mean of the squares of successive normal RR intervals(rMSSD) and pNN50 (successive normal RR intervals>50 ms).
 13. The systemof claim 11, wherein the at least one cardiac parameter is one of HeartRate (HR), Standard Deviation of the normal to normal (SDNN), squareroot of the mean of the squares of successive normal RR intervals(rMSSD) and pNN50 (successive normal RR intervals>50 ms).
 14. The systemof claim 9, wherein the predetermined frequency range is between 0.005Hz-0.012 Hz.
 15. The system of claim 14, wherein the at least onecardiac parameter is one of Heart Rate (HR), Standard Deviation of thenormal to normal (SDNN), square root of the mean of the squares ofsuccessive normal RR intervals (rMSSD) and pNN50 (successive normal RRintervals>50 ms).
 16. The system of claim 9, wherein the at least onecardiac parameter is one of Heart Rate (HR), Standard Deviation of thenormal to normal (SDNN), square root of the mean of the squares ofsuccessive normal RR intervals (rMSSD) and pNN50 (successive normal RRintervals>50 ms).